Light Field Imaging Device and Method for Depth Acquisition and Three-Dimensional Imaging

ABSTRACT

A light field imaging device and method are provided. The device can include a diffraction grating assembly receiving a wavefront from a scene and including one or more diffraction gratings, each having a grating period along a grating axis and diffracting the wavefront to generate a diffracted wavefront. The device can also include a pixel array disposed under the diffraction grating assembly and detecting the diffracted wavefront in a near-field diffraction regime to provide light field image data about the scene. The pixel array has a pixel pitch along the grating axis that is smaller than the grating period. The device can further include a color filter array disposed over the pixel array to spatio-chromatically sample the diffracted wavefront prior to detection by the pixel array. The device and method can be implemented in backside-illuminated sensor architectures. Diffraction grating assemblies for use in the device and method are also disclosed.

TECHNICAL FIELD

The general technical field relates to imaging systems and methods and,more particularly, to a light field imaging device and method for depthacquisition and three-dimensional (3D) imaging.

BACKGROUND

Traditional imaging hardware involves the projection of complexthree-dimensional (3D) scenes onto simplified two-dimensional (2D)planes, forgoing dimensionality inherent in the incident light. Thisloss of information is a direct result of the nature of square-lawdetectors, such as charge-coupled devices (CCD) or complementarymetal-oxide-semiconductor (CMOS) sensor arrays, which can only directlymeasure the time-averaged intensity I of the incident light, not itsphase, φ, or wave vector, k, or angular frequency, w:

I˜<E(t)>; where E(t)=E ₀ cos({right arrow over (k)}·{right arrow over(r)}−ωt+φ).  (1)

Working within this constraint, plenoptic cameras are forced to recoverdepth information through either the comparative analysis of multiplesimultaneously acquired images, complicated machine learning and/orreconstruction techniques, or the use of active illuminators andsensors. Plenoptic cameras generally describe a scene through the“plenoptic function” which parameterizes a light field impingent on anobserver or point by:

P=P(x,y,λ,t,V _(x) ,V _(y) ,V _(z) ,p),  (2)

where the x and y coordinates define a certain image plane at time t,for wavelength A, and polarization angle p, as witnessed by an observerat location (V_(x), V_(y), V_(z)). While they may be single- ormulti-sensor based systems, current plenoptic cameras can rely, atminimum, solely on the intensity of light detected by any given pixel ofa sensor array. More practically, existing solutions, such asstereovision or microlensing, sacrifice overall image quality and sensorfootprint by employing multiple sensors or sensor segmentation toaccommodate the various fields of view required to discern depth.

Random binary occlusion masks and coded apertures are other existingapproaches that provide single-sensor solutions with minimal impact onpackaging or overall footprint. However, despite advances in compressedsensing and non-linear reconstruction techniques, these solutions remainhindered by the massive image dictionaries and computational expenseinvolved.

Time-of-flight and structured-light based techniques actively illuminatea scene with pulsed, patterned, or modulated continuous-wave infraredlight, and determine depth via the full return-trip travel time orsubtle changes in the illuminated light pattern. While these techniquesdo not suffer from image segmentation, they generally require additionalactive infrared emitters and detectors which both increase powerconsumption as well as overall device footprint. Similarly, thesetechniques tend to be sensitive to interfering signals, specularreflections, and ambient infrared light, thus limiting their viabilityoutdoors.

Challenges therefore remain in the field of light field imaging.

SUMMARY

The present description generally relates to light field imagingtechniques for depth mapping and other 3D imaging applications.

In accordance with an aspect, there is provided a light field imagingdevice for capturing light field image data about a scene, the lightfield imaging device including:

-   -   a diffraction grating assembly configured to receive an optical        wavefront originating from the scene, the diffraction grating        assembly including a diffraction grating having a grating axis        and a refractive index modulation pattern having a grating        period along the grating axis, the diffraction grating        diffracting the optical wavefront to generate a diffracted        wavefront; and    -   a pixel array including a plurality of light-sensitive pixels        disposed under the diffraction grating assembly and detecting        the diffracted wavefront as the light field image data, the        pixel array having a pixel pitch along the grating axis that is        smaller than the grating period.

In some implementations, the diffracted wavefront has an intensityprofile along the grating axis, and the pixel array is separated fromthe diffraction grating by a separation distance at which the intensityprofile of the diffracted wavefront has a spatial period thatsubstantially matches the grating period.

In accordance with another aspect, there is provided abackside-illuminated light field imaging device for capturing lightfield image data about a scene, the backside-illuminated light fieldimaging device including:

-   -   a substrate having a front surface and a back surface;    -   a diffraction grating assembly disposed over the back surface of        the substrate and configured to receive an optical wavefront        originating from the scene, the diffraction grating assembly        including a diffraction grating having a grating axis and a        refractive index modulation pattern having a grating period        along the grating axis, the diffraction grating diffracting the        optical wavefront to generate a diffracted wavefront;    -   a pixel array formed in the substrate and including a plurality        of light-sensitive pixels configured to receive through the back        surface and detect as the light field image data the diffracted        wavefront, the pixel array having a pixel pitch along the        grating axis that is smaller than the grating period; and    -   pixel array circuitry disposed under the front surface and        coupled to the pixel array.

In some implementations, the diffracted wavefront has an intensityprofile along the grating axis, and the pixel array is separated fromthe diffraction grating by a separation distance at which the intensityprofile of the diffracted wavefront has a spatial period thatsubstantially matches the grating period.

In accordance with another aspect, there is provided a light fieldimaging device including:

-   -   a diffraction grating assembly including a diffraction grating        having a grating axis and a refractive index modulation pattern        having a grating period along the grating axis; and    -   a pixel array including a plurality of light-sensitive pixels        disposed under the diffraction grating, the pixel array having a        pixel pitch along the grating axis that is smaller than the        grating period.

In accordance with another aspect, there is provided a diffractiongrating assembly for use with an image sensor including a pixel arrayhaving a plurality of light-sensitive pixels to capture light fieldimage data about a scene, the diffraction grating assembly including adiffraction grating having a grating axis and a refractive indexmodulation pattern having a grating period along the grating axis, thegrating period being larger than a pixel pitch of the pixel array alongthe grating axis, the diffraction grating being configured to receiveand diffract an optical wavefront originating from the scene to generatea diffracted wavefront for detection by the light-sensitive pixels asthe light field image data, the diffraction grating assembly beingconfigured to be disposed over the pixel array. In some implementations,the diffraction grating assembly is configured to be separated from thepixel array by a separation distance at which the diffracted wavefronthas an intensity profile along the grating axis with a spatial periodthat substantially matches the grating period.

In accordance with another aspect, there is provided a method ofcapturing light field image data about a scene, the method including:

-   -   diffracting an optical wavefront originating from the scene with        a diffraction grating having a grating period along a grating        axis to generate a diffracted wavefront; and    -   detecting the diffracted wavefront as the light field image data        with a pixel array including a plurality of light-sensitive        pixels disposed under the diffraction grating, the pixel array        having a pixel pitch along the grating axis that is smaller than        the grating period.

In some implementations, the diffracted wavefront has an intensityprofile along the grating axis, and the pixel array is separated fromthe diffraction grating by a separation distance at which the intensityprofile of the diffracted wavefront has a spatial period thatsubstantially matches the grating period.

In accordance with another aspect, there is provided a method ofproviding three-dimensional imaging capabilities to an image sensorviewing a scene and including a pixel array having a plurality oflight-sensitive pixels, the method including:

-   -   disposing a diffraction grating assembly in front of the image        sensor, the diffraction grating assembly including a diffraction        grating having a grating axis and a grating period along the        grating axis, the grating period being larger than a pixel pitch        of the pixel array along the grating axis;    -   receiving and diffracting an optical wavefront originating from        the scene with the diffraction grating to generate a diffracted        wavefront; and    -   detecting the diffracted wavefront with the light-sensitive        pixels.

In some implementations, disposing the diffraction grating assembly infront of the image sensor includes positioning the diffraction gratingassembly at a separation distance from the pixel array at which thediffracted wavefront has an intensity profile along the grating axiswith a spatial period that substantially matches the grating period.

In some implementations, the light field imaging device can include anarray of light-sensitive elements; an array of color filters overlyingand aligned with the array of photosensitive elements such that eachcolor filter covers at least one of the light-sensitive elements, thecolor filters being spatially arranged according to a mosaic colorpattern; and a diffraction grating structure extending over the array ofcolor filters.

In some implementations, the light field imaging device can include adiffraction grating structure exposed to an optical wavefront incidentfrom a scene, the diffraction grating structure diffracting the opticalwavefront to produce a diffracted wavefront; an array of color filtersspatially arranged according to a mosaic color pattern, the array ofcolor filters extending under the diffraction grating structure andspatio-chromatically filtering the diffracted wavefront according to themosaic color pattern to produce a filtered wavefront including aplurality of spatially distributed wavefront components; and an array oflight-sensitive elements detecting the filtered wavefront as light fieldimage data, the array of light-sensitive elements underlying and beingaligned with the array of color filters such that each light-sensitiveelement detects at least a corresponding one of the spatiallydistributed wavefront components.

In some implementations, the method can include diffracting an opticalwavefront incident from a scene to produce a diffracted wavefront;filtering the diffracted wavefront through an array of color filtersspatially arranged according to a mosaic color pattern, therebyobtaining a filtered wavefront including a plurality of spatiallydistributed wavefront components; and detecting the filtered wavefrontas light field image data with an array of light-sensitive elementsunderlying and aligned with the array of color filters such that eachlight-sensitive element detects at least part of a corresponding one ofthe spatially distributed wavefront components.

In some implementations, the method can include diffracting an opticalwavefront incident from a scene to produce a diffracted wavefront;spectrally and spatially filtering the diffracted wavefront to produce afiltered wavefront including a plurality of spatially distributed andspectrally filtered wavefront components; and detecting as light fieldimage data the plurality of spatially distributed and spectrallyfiltered wavefront components at a plurality of arrayed light-sensitiveelements.

Other features and advantages of the present description will becomemore apparent upon reading of the following non-restrictive descriptionof specific embodiments thereof, given by way of example only withreference to the appended drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic perspective view of a light field imaging device,in accordance with a possible embodiment.

FIG. 2 is a schematic partially exploded perspective view of the lightfield imaging device of FIG. 1.

FIG. 3 is a schematic partially exploded perspective view of a lightfield imaging device, in accordance with another possible embodiment,where each color filter overlies a 2×2 block of light-sensitive pixels.

FIG. 4 is a schematic perspective view of a light field imaging device,in accordance with another possible embodiment, where the light fieldimaging device is configured for monochrome imaging applications.

FIG. 5 is a schematic partially exploded perspective view of the lightfield imaging device of FIG. 4.

FIG. 6 is a schematic partially exploded perspective view of the lightfield imaging device, in accordance with another possible embodiment,where the light field imaging device includes a microlens array on topof the color filter array.

FIG. 7 is a schematic partially exploded side view of a light fieldimaging device, in accordance with another possible embodiment, wherethe propagation of a wavefront of light through the device isschematically depicted. The light field imaging device of FIG. 7 issuitable for monochrome imaging applications.

FIGS. 8A to 8C are schematic partially exploded side views of threeother possible embodiments of a light field imaging device, where thepropagation of a wavefront of light through the device is schematicallydepicted. In FIG. 8A, each light-sensitive pixel is vertically alignedwith a transition between one ridge and one groove. In FIG. 8B, theratio of the grating period to the pixel pitch along the grating axis isequal to four. In FIG. 8C, the duty cycle of the diffraction grating isdifferent from 50%.

FIGS. 9A and 9B are schematic partially transparent top views of twoother possible embodiments of a light field imaging device, where thegrating axis of the diffraction grating is oblique to either of the twoorthogonal pixel axes.

FIG. 10 is a schematic partially exploded side view of a light fieldimaging device, in accordance with another possible embodiment, wherethe propagation of a wavefront of light through the device isschematically depicted. The light field imaging device of FIG. 10 issuitable for color imaging applications.

FIG. 11 is a schematic perspective view of a light field imaging device,in accordance with another possible embodiment, where the diffractinggrating assembly includes two sets of orthogonally oriented diffractinggratings arranged to alternate in both rows and columns to define acheckerboard pattern.

FIGS. 12A to 12C illustrate alternative embodiments of diffractiongrating assemblies including a plurality of diffraction gratingsarranged in a two-dimensional array.

FIG. 13 is a schematic perspective view of a light field imaging device,in accordance with another possible embodiment, where the diffractinggrating assembly includes a plurality of diffraction gratings forming anarray of color filters, each of which embodied by a respective one ofthe diffraction gratings.

FIG. 14 is a schematic side view of a light field imaging device, inaccordance with another possible embodiment, where the light fieldimaging device includes dispersive optics disposed in front of thediffraction grating assembly to spatio-spectrally spread the opticalwavefront originating from the scene prior to it reaching thediffraction grating assembly.

FIG. 15 is a schematic side view of a light field imaging device in afrontside illumination configuration, in accordance with anotherpossible embodiment.

FIG. 16 is a schematic side view of a light field imaging device in abackside illumination configuration, in accordance with another possibleembodiment.

FIG. 17 is a schematic perspective view of a diffraction gratingassembly for use in an image sensor including a pixel array having aplurality of light-sensitive pixels to capture light field image dataabout a scene, in accordance with a possible embodiment.

FIG. 18 is a flow diagram of a method of capturing light field imagedata about a scene, in accordance with a possible embodiment.

FIG. 19 is a flow diagram of a method of providing 3D imagingcapabilities to an image sensor viewing a scene and including an arrayof light-sensitive pixels, in accordance with a possible embodiment.

DETAILED DESCRIPTION

In the present description, similar features in the drawings have beengiven similar reference numerals, and, to not unduly encumber thefigures, some elements may not be indicated on some figures if they werealready identified in a preceding figure. It should also be understoodthat the elements of the drawings are not necessarily depicted to scale,since emphasis is placed upon clearly illustrating the elements andstructures of the present embodiments.

In the present description, and unless stated otherwise, the terms“connected”, “coupled” and variants and derivatives thereof refer to anyconnection or coupling, either direct or indirect, between two or moreelements. The connection or coupling between the elements may bemechanical, optical, electrical, operational or a combination thereof.It will also be appreciated that positional descriptors and other liketerms indicating the position or orientation of one element with respectto another element are used herein for ease and clarity of descriptionand should, unless otherwise indicated, be taken in the context of thefigures and should not be considered limiting. It will be understoodthat such spatially relative terms are intended to encompass differentorientations in use or operation of the present embodiments, in additionto the orientations exemplified in the figures. More particularly, it isto be noted that in the present description, the terms “over” and“under” in specifying the relative spatial relationship of two elementsdenote that the two elements can be either in direct contact with eachother or separated from each other by one or more intervening elements.

In the present description, the terms “a”, “an” and “one” are defined tomean “at least one”, that is, these terms do not exclude a plural numberof items, unless specifically stated otherwise.

The present description generally relates to light field imagingtechniques for acquiring light field information or image data about anoptical wavefront emanating from a scene. In accordance with variousaspects, the present description relates to a light field imaging devicefor capturing light field image data about a scene, for example abackside-illuminated light field imaging device; a diffraction gratingassembly for use with an image sensor to obtain light field image dataabout a scene; a method of capturing light field image data about ascene; and a method of providing three-dimensional (3D) imagingcapabilities to an image sensor array viewing a scene.

In some implementations, the present techniques enable the specificmanipulation and comparison of the chromatic dependence of diffractionby means of one or many diffractive optical elements paired with anappropriate chromatic encoding mechanism, as well as its use in 3Dimaging. In some implementations, the light field imaging devices andmethods disclosed herein are sensitive to not only the intensity andangle of incidence of an optical wavefront originating from anobservable scene, but also the wavelength, through a specificspatio-spectral subsampling of a generated interference pattern allowingfor direct measurement of the chromatic dependence of diffraction. Lightfield information or image data, can include information about not onlythe intensity of the optical wavefront emanating from an observablescene, but also other light field parameters including, withoutlimitation, the angle of incidence, the phase, the wavelength and thepolarization of the optical wavefront. Therefore, light field imagingdevices, for example depth cameras, can acquire more information thantraditional cameras, which typically record only light intensity. Theimage data captured by light field imaging devices can be used orprocessed in a variety of ways to provide multiple functions including,but not limited to, 3D depth map extraction, 3D surface reconstruction,image refocusing, and the like. Depending on the application, the lightfield image data of an observable scene can be acquired as one or morestill images or as a video stream.

The present techniques can be used in imaging applications that requireor can benefit from enhanced depth sensing and other 3D imagingcapabilities, for example to allow a user to change the focus, the pointof view and/or the depth of field of a captured image of a scene. Thepresent techniques can be applied to or implemented in various types of3D imaging systems and methods including, without limitation, lightfield imaging applications using plenoptic descriptions, rangingapplications through the comparative analysis of the chromaticdependence of diffraction, and single-sensor single-image depthacquisition applications. Non-exhaustive advantages and benefits ofcertain implementations of the present techniques can include:compatibility with passive sensing modalities that employ less power toperform their functions; compatibility with single-sensor architectureshaving reduced footprint; enablement of depth mapping functions whilepreserving 2D performance; simple and low-cost integration into existingimage sensor hardware and manufacturing processes; compatibility withconventional CMOS and CCD image sensors; and elimination of the need formultiple components, such as dual cameras or cameras equipped withactive lighting systems for depth detection.

In the present description, the terms “light” and “optical” are used torefer to radiation in any appropriate region of the electromagneticspectrum. More particularly, the terms “light” and “optical” are notlimited to visible light, but can also include invisible regions of theelectromagnetic spectrum including, without limitation, the terahertz(THz), infrared (IR) and ultraviolet (UV) spectral bands. In someimplementations, the terms “light” and “optical” can encompasselectromagnetic radiation having a wavelength ranging from about 175nanometers (nm) in the deep ultraviolet to about 300 micrometers (μm) inthe terahertz range, for example from about 400 nm at the blue end ofthe visible spectrum to about 1550 nm at telecommunication wavelengths,or between about 400 nm and about 650 nm to match the spectral range oftypical red-green-blue (RGB) color filters. Those skilled in the artwill understand, however, that these wavelength ranges are provided forillustrative purposes only and that the present techniques may operatebeyond this range.

In the present description, the terms “color” and “chromatic”, andvariants and derivatives thereof, are used not only in their usualcontext of human perception of visible electromagnetic radiation (e.g.,red, green and blue), but also, and more broadly, to describe spectralcharacteristics (e.g., diffraction, transmission, reflection,dispersion, absorption) over any appropriate region of theelectromagnetic spectrum. In this context, and unless otherwisespecified, the terms “color” and “chromatic” and their derivatives canbe used interchangeably with the term “spectral” and its derivatives.

Light Field Imaging Device Implementations

Referring to FIGS. 1 and 2, there is provided a schematic representationof an exemplary embodiment of a light field imaging device 20 forcapturing light field or depth image data about an observable scene 22.In the present description, the term “light field imaging device”broadly refers to any image capture device capable of acquiring an imagerepresenting a light field or wavefront emanating from a scene andcontaining information about not only light intensity at the imageplane, but also other light field parameters such as, for example, thedirection from which light rays enter the device and the spectrum of thelight field. It is to be noted that in the present description, the term“light field imaging device” can be used interchangeably with terms suchas “light field camera”, “light field imager”, “light field imagecapture device”, “depth image capture device”, “3D image capturedevice”, and the like.

In the illustrated embodiment, the light field imaging device 20includes a diffraction grating assembly or structure 24 configured toreceive an optical wavefront 26 originating from the scene 22. Thediffraction grating assembly 24 can include at least one diffractiongrating 28, each of which having a grating axis 30 and a refractiveindex modulation pattern 32 having a grating period 34 along the gratingaxis 30. In FIGS. 1 and 2, the diffraction grating assembly 24 includesa single diffraction grating 28, but as described below, in otherembodiments the diffraction grating assembly can include more than onediffraction grating. The diffraction grating 28 is configured todiffract the incoming optical wavefront 26, thereby generating adiffracted wavefront 36. The diffraction grating 28 in FIGS. 1 and 2 isused in transmission since the incident optical wavefront 26 and thediffracted wavefront 36 lie on opposite sides of the diffraction grating28.

Referring still to FIGS. 1 and 2, the light field imaging device 20 alsoincludes a pixel array 38 comprising a plurality of light-sensitivepixels 40 disposed under the diffraction grating assembly 24 andconfigured to detect the diffracted wavefront 36 as the light fieldimage data about the scene 22. In color implementations, the light fieldimaging device 20 can also include a color filter array 42 disposed overthe pixel array 38. The color filter array 42 includes a plurality ofcolor filters 44 arranged in a mosaic color pattern, each of whichfilters incident light by wavelength to capture color information at arespective location in the color filter array 42. The color filter array42 is configured to spatially and spectrally filter the diffractedwavefront 36 according to the mosaic color pattern prior to detection ofthe diffracted wavefront 36 by the plurality of light-sensitive pixels40. Therefore, as mentioned above, by providing a color filter array toperform a direct spatio-chromatic subsampling of the diffractedwavefront generated by the diffraction grating assembly prior to itsdetection by the pixel array, the light field imaging device can besensitive to not only the angle and intensity of an incident wavefrontof light, but also its spectral content.

It is to be noted that a color filter array need not be provided in someapplications, for example for monochrome imaging. It is also to be notedthat the wavefront detected by the light-sensitive pixels will begenerally referred to as a “diffracted wavefront” in both monochrome andcolor implementations, although in the latter case, the terms “filteredwavefront” or “filtered diffracted wavefront” may, in some instances, beused to denote the fact that the diffracted wavefront generated by thediffraction grating assembly is both spatially and spectrally filteredby the color filter array prior to detection by the underlying pixelarray. It is also to be noted that in some implementations where a colorfilter array is not provided, it may be envisioned that the diffractiongrating could act as a color filter. For example, the diffractiongrating could include a grating substrate with a top surface having therefractive index modulation pattern formed thereon, the gratingsubstrate including a spectral filter material or region configured tospectrally filter the diffracted wavefront according to wavelength priorto detection of the diffracted wavefront by the plurality oflight-sensitive pixels. For example, the spectral filter material orregion could act as one of a red pass filter, a green pass filter and ablue pass filter.

Depending on the application or use, embodiments of the light fieldimaging device can be implemented using various image sensorarchitectures and pixel array configurations. For example, in someimplementations, the light field imaging device can be implementedsimply by adding or coupling a diffraction grating assembly on top of analready existing image sensor including a pixel array and, incolor-based applications, a color filter array. For example, theexisting image sensor can be a conventional 2D CMOS or CCD imager.However, in other implementations, the light field imaging device can beimplemented and integrally packaged as a separate, dedicated and/orcustom-designed device incorporating therein all or most of itscomponents (e.g., diffraction grating assembly, pixel array, colorfilter array).

More detail regarding the structure, configuration and operation of thecomponents introduced in the preceding paragraphs as well as otherpossible components of the light field imaging device will be describedbelow.

In the embodiment illustrated in FIGS. 1 and 2, the diffraction grating28 includes a grating substrate 46 extending over the color filter array42. The grating substrate 46 has a top surface 48, on which is formedthe periodic refractive index modulation pattern 32, and a bottomsurface 50. The grating substrate 46 is made of a material that istransparent, or sufficiently transparent, in the spectral operatingrange to permit the diffracted wavefront 36 to be transmittedtherethrough. Non-limiting examples of such material include siliconoxide (SiOx), polymers, colloidal particles, SU-8 photoresist, glasses.For example, in some implementations the diffraction grating 28 can beconfigured to diffract the optical wavefront 26 in a waveband rangingfrom about 400 nm to about 1550 nm.

As known in the art, diffraction occurs when a wavefront, whetherelectromagnetic or otherwise, encounters a physical object or arefractive-index perturbation. The wavefront tends to bend around theedges of the object. Should a wavefront encounter multiple objects,whether periodic or otherwise, the corresponding wavelets may interferesome distance away from the initial encounter as demonstrated by Young'sdouble slit experiment. This interference creates a distinct pattern,referred to as a “diffraction pattern” or “interference pattern”, as afunction of distance from the original encounter, which is sensitive tothe incidence angle and the spectral content of the wavefront, and thegeneral size, shape, and relative spatial relationships of theencountered objects. This interference can be described through theevolving relative front of each corresponding wavelet, as described bythe Huygens-Fresnel principle.

In the present description, the term “diffraction grating”, or simply“grating”, generally refers to a periodic structure having periodicallymodulated optical properties (e.g., a refractive index modulationpattern) that spatially modulates the amplitude and/or the phase of anoptical wavefront incident upon it. A diffraction grating can include aperiodic arrangement of diffracting elements (e.g., alternating ridgesand grooves) whose spatial period—the grating period—is nearly equal toor slightly longer than the wavelength of light. An optical wavefrontcontaining a range of wavelengths incident on a diffraction gratingwill, upon diffraction, have its amplitude and/or phase modified, and,as a result, a space- and time-dependent diffracted wavefront isproduced. In general, a diffracting grating is spectrally dispersive sothat each wavelength of an input optical wavefront will be outputtedalong a different direction. However, diffraction gratings exhibiting asubstantially achromatic response over an operating spectral range existand can be used in some implementations. For example, in someimplementations, the diffraction grating can be achromatic in thespectral range of interest and be designed for the center wavelength ofthe spectral range of interest. More particularly, in the case of aBayer patterned color filter array, the diffraction grating can beoptimized for the green channel, that is, around a center wavelength ofabout 532 nm. It is to be noted that when the diffraction grating isachromatic, it is the mosaic color patter of the color filter array thatprovides the chromatic sub-sampling of the diffraction pattern of thediffracted wavefront.

Depending on whether the diffracting elements forming the diffractiongrating are transmitting or reflective, the diffraction grating will bereferred to as a “transmission grating” or a “reflection grating”. Inthe embodiments disclosed in the present description, the diffractinggratings are transmission gratings, although the use of reflectiongratings is not excluded a priori. Diffraction gratings can also beclassified as “amplitude gratings” or “phase gratings”, depending on thenature of diffracting elements. In amplitude gratings, the perturbationsto the initial wavefront caused by the grating are the result of adirect amplitude modulation, while in phase gratings, theseperturbations are the result of a specific modulation of the relativegroup-velocity of light caused by a periodic variation of the refractiveindex of the grating material. In the embodiments disclosed in thepresent description, the diffracting gratings are phase gratings,although amplitude gratings can be used in other embodiments.

In the illustrated embodiment of FIGS. 1 and 2, the diffraction grating28 is a phase grating, more specifically a binary phase grating forwhich the refractive index modulation pattern 32 includes a series ofridges 52 periodically spaced-apart at the grating period 34,interleaved with a series of grooves 54 also periodically spaced-apartat the grating period 34. The spatial profile of the refractive indexmodulation pattern 32 thus exhibits a two-level step function, orsquare-wave function, for which the grating period 34 corresponds to thesum of the width, along the grating axis 30, of one ridge 52 and oneadjacent groove 54. In some implementations, the grating period 34 canrange from about 1 μm to about 20 μm, although other values are possiblein other implementations. In the illustrated embodiment of FIGS. 1 and2, the grooves 54 are empty (i.e., they are filled with air), but theycould alternatively be filled with a material having a refractive indexdifferent from that of the ridge material. Also, depending on theapplication, the diffraction grating 28 can have a duty cyclesubstantially equal to or different from 50%, the duty cycle beingdefined as the ratio of the ridge width to the grating period 34.Another parameter of the diffraction grating 28 is the step height 56,that is, the difference in level between the ridges 52 and the grooves54. For example, in some implementations the step height 56 can rangefrom about 0.2 μm to about 1 μm. It is to be noted that in someimplementations, the step height 56 can be selected so that thediffraction grating 28 causes a predetermined optical path differencebetween adjacent ridges 52 and grooves 54. For example, the step height56 can be controlled to provide, at a given wavelength and angle ofincidence of the optical wavefront (e.g. its center wavelength), ahalf-wave optical path difference between the ridges and the grooves. Ofcourse, other optical path difference values can be used in otherimplementations.

It is to be noted that while the diffraction grating 28 in theembodiment of FIGS. 1 and 2 is a linear, or one-dimensional, binaryphase grating consisting of alternating sets of parallel ridges 52 andgrooves 54 forming a square-wave refractive index modulation pattern 32,other embodiments can employ different types of diffraction gratings.For example, other implementations can use diffraction gratings where atleast one among the grating period, the duty cycle and the step heightis variable; diffraction gratings with non-straight featuresperpendicular to the grating axis; diffraction gratings having moreelaborate refractive index profiles; 2D diffraction gratings; and thelike. It will be understood that the properties of the diffractedwavefront can be tailored by proper selection of the grating parameters.More detail regarding the operation of the diffraction grating and itspositioning relative and optical coupling to the other components of thelight field imaging device will be described further below.

Referring still to FIGS. 1 and 2, as mentioned above, the pixel array 38includes a plurality of light-sensitive pixels 40 disposed under thecolor filter array 42, which is itself disposed under the diffractiongrating assembly 24. The term “pixel array” refers generally to a sensorarray made up of a plurality of photosensors, referred to herein as“light-sensitive pixels” or simply “pixels”, which are configured todetect electromagnetic radiation incident thereonto from an observablescene and to generate an image of the scene, typically by converting thedetected radiation into electrical data. In the present techniques, theelectromagnetic radiation that is detected by the light-sensitive pixels40 as light field image data corresponds to an optical wavefront 26incident from the scene 22, which has been diffracted and, optionally,spatio-chromatically filtered, prior to reaching the pixel array 38. Thepixel array 38 can be embodied by a CMOS or a CCD image sensor, butother types of photodetector arrays (e.g., charge injection devices orphotodiode arrays) could alternatively be used. As mentioned above, thepixel array 38 can be configured to detect electromagnetic radiation inany appropriate region of the spectrum. Depending on the application,the pixel array 38 may be configured according to either a rolling orglobal shutter readout design. The pixel array 38 may further be part ofa stacked, backside, or frontside illumination sensor architecture, asdescribed in greater detail below. The pixel array 38 may be of anystandard or non-standard optical format, for example, but not limitedto, 4/3″, 1″, ⅔″, 1/1.8″, ½″, 1.27″, ⅓″, 1/3.2″, 1/3.6″, 35 mm, and thelike. The pixel array 38 may also include either a contrast or aphase-detection autofocus mechanism and their respective pixelarchitectures. It is to be noted that in the present description, theterm “pixel array” can be used interchangeably with terms such as“photodetector array”, “photosensor array”, “imager array”, and thelike.

Each light-sensitive pixel 40 of the pixel array 38 can convert thespatial part of the diffracted wavefront 36 incident upon it intoaccumulated charge, the amount of which is proportional to the amount oflight collected and registered by the pixel 40. Each light-sensitivepixel 40 can include a light-sensitive surface and associated pixelcircuitry for processing signals at the pixel level and communicatingwith other electronics, such as a readout unit. Those skilled in the artwill understand that various other components can be integrated into thepixel circuitry of each pixel. In general, the light-sensitive pixels 40can be individually addressed and read out.

Referring still to FIGS. 1 and 2, the light-sensitive pixels 40 can bearranged into a rectangular grid of rows and columns defined by twoorthogonal pixel axes 58, 60, the number of rows and columns definingthe resolution of the pixel array 38. For example, in someimplementations, the pixel array 38 can have a resolution of at least 16pixels, although a wide range of other resolution values, including upto 40 megapixels or more, can be used in other embodiments. It is to benoted that while the light-sensitive pixels 40 are organized into a 2Darray in the embodiment of FIGS. 1 and 2, they may alternatively beconfigured as a linear array in other embodiments. It is also to benoted that while the light-sensitive pixels 40 are square in theembodiment of FIGS. 1 and 2, corresponding to a pixel aspect ratio of1:1, other pixel aspect ratio values can be used in other embodiments.

The pixel array 38 can also be characterized by a pixel pitch 62. In thepresent description, the term “pixel pitch” generally refers to thespacing between the individual pixels 40 and is typically defined as thecenter-to-center distance between adjacent pixels 40. Depending on thephysical arrangement of the pixel array 38, the pixel pitch 62 along thetwo orthogonal pixel axes 58, 60 may or may not be the same. It is to benoted that a pixel pitch can also be defined along an arbitrary axis,for example along a diagonal axis oriented at 45° with respect to thetwo orthogonal pixel axes 58, 60. It is also to be noted that, in thepresent techniques, a relevant pixel pitch 62 is the one along thegrating axis 30 of the overlying diffraction grating 28, as depicted inFIGS. 1 and 2. As described in greater detail below, the grating period34 of the diffraction grating 28 is selected to be larger than the pixelpitch 62 of the pixel array 38 along the grating axis 30. For example,in some implementations the pixel pitch 62 along the grating axis 30 canrange from 1 μm or less to 10 μm, although different pixel pitch valuescan be used in other implementations.

In the present description, the term “pixel data” refers to the imageinformation captured by each individual pixel and can include intensitydata indicative of the total amount of optical energy absorbed by eachindividual pixel over an integration period. Combining the pixel datafrom all the pixels 40 yields light field image data about the scene 22.In the present techniques, because the optical wavefront 26 incidentfrom the scene 22 is diffracted and, possibly, spatially and spectrallyfiltered prior to detection, the light field image data can provideinformation about not only the intensity of the incident wavefront 26,but also other light field parameters such as its angle of incidence,phase and spectral content. More particularly, it will be understoodthat the present techniques can allow recovery or extraction of depth orother light field information from the intensity-based diffractionpattern captured by the pixel array 38, as described further below.

Referring still to FIGS. 1 and 2, the color filter array 42 is spatiallyregistered with the pixel array 38, such that each color filter 44 isoptically coupled to a corresponding one of the light-sensitive pixels40. That is, each color filter 44 covers a single light-sensitive pixel40, such that there is a one-to-one relationship, or mapping, betweenthe color filters 44 and the light-sensitive pixels 40. However, inother implementations, each color filter can be optically coupled to atleast two corresponding ones of the plurality of light-sensitive pixels.For example, turning briefly to FIG. 3, there is shown anotherembodiment of a light field imaging device 20 in which each color filter44 of the color filter array 42 overlies a group or subset oflight-sensitive pixels 40, namely a 2×2 block of light-sensitive pixels40. In both the embodiment of FIGS. 1 and 2 and the embodiment of FIG.3, the color filter array 42 and the pixel array 38 together enable thedirect spatio-chromatic sampling of the diffracted wavefront produced bythe overlying diffraction grating assembly 24, as detailed and explainedbelow.

As mentioned above regarding the terms “color” and “chromatic”, termssuch as “color filter” and “color filtering” are to be understood asbeing equivalent to “spectral filter” and “spectral filtering” in anyappropriate spectral range of the electromagnetic spectrum, and not onlywithin the visible range. Depending on the application, the colorfilters can achieve spectral filtering through absorption of unwantedspectral components, for example using dye-based color filters, althoughother filtering principles may be used without departing from the scopeof the present techniques.

Returning to FIGS. 1 and 2, the color filters 44 are physicallyorganized according to a mosaic color pattern or configuration. In someimplementations, each color filter 44 is one of a red pass filter, agreen pass filter and a blue pass filter. For example, in theillustrated embodiment, the mosaic color pattern of the color filterarray 42 is a Bayer pattern, in which the color filters arranged in acheckerboard pattern with rows of alternating red (R) and green (G)filters are interleaved with rows of alternating green (G) and blue (B)filters. As known in the art, a Bayer pattern contains twice as manygreen filters as red or blue filters, such that the green component ofthe mosaic color pattern is more densely sampled than red and bluecomponents. In alternative implementations, the mosaic color pattern canbe embodied by more elaborate Bayer-type patterns, for exampleBayer-type patterns with an n-pixel unit cell, where n is an integergreater than 4. Of course, the present techniques are not limited toBayer-type patterns, but can be applied to any appropriate mosaic colorpattern including, but not limited to, RGB, RGB-IR, RGB-W, CYGM, CYYM,RGBE, RGBW #1, RGBW #2, RGBW #3, and monochrome. It is to be noted thatin some implementations, the color filter array 42 may be extendedbeyond the standard visible Bayer pattern to include hyperspectralimaging and filtering techniques or interferometric filteringtechniques. In such embodiments, the design of the diffraction grating28 (e.g., the grating period 34) can be adjusted to accommodate theincreased spectral sampling range.

Referring now to FIGS. 4 and 5, there is shown another embodiment of alight field imaging device 20, which is suitable for monochrome imagingapplications. This embodiment shares many features with the embodimentdescribed above and illustrated in FIGS. 1 and 2, insofar as itgenerally includes a diffraction grating assembly 24 including at leastone diffraction grating 28 and disposed over a pixel array 38 includinga plurality of light-sensitive pixels 40. These components can generallybe similar in terms of structure and operation to like components of theembodiment of FIGS. 1 and 2. The light field imaging device 20 of FIGS.4 and 5 differs from that of FIGS. 1 and 2 mainly in that it does notinclude a color filter array disposed between the diffraction gratingassembly 24 and the pixel array 38. As a result, the light-sensitivepixels 40 directly detect the diffracted wavefront 36 transmitted by thediffraction grating 28.

Referring to FIG. 6, there is shown another embodiment of a light fieldimaging device 20, which shares similar features with the embodiment ofFIGS. 4 and 5, but differs in that it further includes a microlens array64 disposed over the pixel array 38 and including a plurality ofmicrolenses 66. Each microlens 66 is optically coupled to acorresponding one of the light-sensitive pixels 40 and is configured tofocus the spatial part of the diffracted wavefront 36 incident upon itonto its corresponding light-sensitive pixel 40. It is to be noted thatin embodiments where an array of color filters is provided, such as inFIGS. 1 and 2, the microlens array would be disposed over the colorfilter array such that each microlens would be optically coupled to acorresponding one of the color filters. In some variants, the lightimaging device may also include an anti-reflection coating (not shown)provided over the pixel array 38.

Referring now to FIG. 7, there is shown a schematic partially explodedside view of an embodiment of a light field imaging device 20 suitablefor monochrome imaging applications. The light field imaging device 20shares similarities with the one shown in FIGS. 4 and 5, in that itincludes a diffraction grating 28 disposed on top of a pixel array 38 oflight-sensitive pixels 40. The diffraction grating 28 is a binary phasetransmission grating having a duty cycle of 50% and a periodicrefractive index modulation pattern 32 consisting of alternating sets ofridges 52 and grooves 54. FIG. 7 also depicts schematically thepropagation of light through the device 20. In operation, the lightfield imaging device 20 has a field of view encompassing an observablescene 22. The diffraction grating 28 receives an optical wavefront 26(solid line) incident from the scene 22 on its input side, and diffractsthe optical wavefront 26 to generate, on its output side, a diffractedwavefront 36 (solid line) that propagates toward the pixel array 38 fordetection thereby. For simplicity, the incoming optical wavefront 26 inFIG. 7 corresponds to the wavefront of a plane wave impinging on thediffraction grating 28 at normal incidence. However, the presenttechniques can be implemented for an optical wavefront of arbitraryshape incident on the diffraction grating 28 at an arbitrary anglewithin the field of view of the light field imaging device.

Referring still to FIG. 7, the diffracted wavefront 36 can becharacterized by a diffraction pattern whose form is a function of thegeometry of the diffraction grating 28, the wavelength and angle ofincidence of the optical wavefront 26, and the position of theobservation plane, which corresponds to the light-receiving surface 68of the pixel array 38. In the observation plane, the diffraction patternof the diffracted wavefront 36 can be characterized by a spatiallyvarying intensity profile 70 along the grating axis 30 in thelight-receiving surface 68 of the pixel array 38. It is to be noted thatin FIG. 7, the grating axis 30 is parallel to the pixel axis 58.

In the present techniques, the diffraction grating 28 and the pixelarray 38 are disposed relative to each other such that thelight-receiving surface 68 of the pixel array 38 is positioned in thenear-field diffraction region, or simply the near field, of thediffraction grating 28. In the near-field diffraction regime, theFresnel diffraction theory can be used to calculate the diffractionpattern of waves passing through a diffraction grating. Unlike thefar-field Fraunhofer diffraction theory, Fresnel diffraction accountsfor the wavefront curvature, which allows calculation of the relativephase of interfering waves. Similarly, when detecting the diffractedirradiance pattern within a few integer multiples of the wavelength witha photosensor or another imaging device of the same dimensional order asthe grating, higher order-diffractive effects tend to be limited simplyby spatial sampling. To detect the diffracted wavefront 36 in the nearfield, the present techniques can involve maintaining a sufficientlysmall separation distance 72 between the top surface 48 of thediffraction grating 28, where refractive index modulation pattern 32 isformed and diffraction occurs, and the light-receiving surface 68 of theunderlying pixel array 38, where the diffracted wavefront 36 isdetected. In some implementations, this can involve selecting theseparation distance 72 to be less than about ten times a centerwavelength of the optical wavefront 26. In some implementations, theseparation distance 72 can range between about 0.5 μm and about 20 μm,for example between 0.5 μm and about 8 μm if the center wavelength ofthe optical wavefront lies in the visible range.

In the near-field diffraction regime, the intensity profile 70 of thediffracted wavefront 36 produced by a periodic diffraction grating 28generally has a spatial period 74 that substantially matches the gratingperiod 34 of the diffraction grating 28 as well as a shape thatsubstantially matches the refractive index modulation pattern 32 of thediffraction grating 28. For example, in the illustrated embodiment, thediffraction pattern of the diffracted wavefront 36 detected by thelight-sensitive pixels 40 of the pixel array 38 has a square-wave, ortwo-step, intensity profile 70 that substantially matches that of therefractive index modulation pattern 32 of the binary phase diffractiongrating 28. In the present description, the term “match” and derivativesthereof should be understood to encompass not only an “exact” or“perfect” match between the intensity profile 70 of the detecteddiffracted wavefront 36 and the periodic refractive index modulationpattern 32 of the diffraction grating 28, but also a “substantial”,“approximate” or “subjective” match. The term “match” is thereforeintended to refer herein to a condition in which two features are eitherthe same or within some predetermined tolerance of each other.

Another feature of near-field diffraction by a periodic diffractiongrating is that upon varying the angle of incidence 76 of the incomingoptical wavefront 26 on the diffraction grating 28, the intensityprofile 70 of the diffracted wavefront 36 is laterally shifted along thegrating axis 30, but substantially retains its period 74 and shape, ascan be seen from the comparison between solid and dashed wavefront linesin FIG. 7. It will be understood that in some implementations, theseparation distance between the diffraction grating 28 and the pixelarray 38 can be selected to ensure the spatial shift experienced by theintensity profile 70 of the diffracted wavefront 36 remains less thanthe grating period 34 as the angle of incidence 76 of the opticalwavefront 26 is varied across the angular span of the field of view ofthe light field imaging device 20. Otherwise, ambiguity in the angle ofincidence 76 of the optical wavefront 26 can become an issue. Forexample, consider for illustrative purposes, a light field imagingdevice 20 whose field of view has an angular span of ±20° and in whichvarying the angle of incidence 76 of the incoming optical wavefront 26by 10° produces a spatial shift of the intensity profile 70 of thediffracted wavefront 36 equal to the grating period 34. In such a case,light incident on the diffraction grating 34 with an incidence angle of,for example, +2° would be undistinguishable, from phase informationalone, from light incidence on the diffraction grating 34 with anincidence angle of +12°.

It is also to be noted that upon being optically coupled to anunderlying pixel array 38, the diffraction grating 28 convolves lightsphase information with a standard 2D image, so that the intensityprofile 70 of the diffraction pattern of the detected diffractedwavefront 36 can generally be written as a modulated functionI˜I_(mod)(depth info)×I_(base) (2D image) including a modulatingcomponent I_(mod) and a base component I_(base). The base componentI_(base) represents the non-phase-dependent optical wavefront that wouldbe detected by the pixel array 38 if there were no diffraction grating28 in front of it. In other words, detecting the base component I_(base)alone would allow a conventional 2D image of the scene 22 to beobtained. Meanwhile, the modulating component I_(mod), which isgenerally small compared to the base component I_(base) (e.g., ratio ofI_(mod) to I_(base) ranging from about 0.1 to about 0.3), is a directresult of the phase of the incident optical wavefront 26, so that anyedge or slight difference in incidence angle will manifest itself as aperiodic electrical response spatially sampled across the pixel array38. It will be understood that the sensitivity to the angle of incidence76 of the optical wavefront 26, and therefore the angular resolution ofthe light field imaging device 20, will generally depend on the specificdesign of the diffraction grating 28.

Referring still to FIG. 7, as mentioned above, in the presenttechniques, the pixel array 38 has a pixel pitch 62 along the gratingaxis 30 that is smaller than the grating period 34 of the diffractiongrating 28. This means that when the light-receiving surface 68 of thepixel array 38 is in the near field of the diffracting grating 28, thepixel pitch 62 of the pixel array 38 along the grating axis 30 is alsosmaller than the spatial period 74 of the intensity profile 70 along thegrating axis 30 of the detected diffracted wavefront 36. It will beunderstood that when this condition is fulfilled, a complete period ofthe intensity profile 70 of the detected diffracted wavefront 36 will besampled by at least two adjacent pixel banks of the pixel array 38, eachof these pixel banks sampling a different spatial part of the intensityprofile 70 over a full cycle. In the present description, the term“pixel bank” refers to a group of light-sensitive pixels of the pixelarray that are arranged along a line which is perpendicular to thegrating axis of the overlying diffraction grating. That is, two adjacentpixel banks are separated from each other by a distance corresponding tothe pixel pitch along the grating axis. For example, in FIG. 7, eachpixel bank of the pixel array 38 extends parallel to the pixel axis 60oriented perpendicular to the plane of the page.

It will be understood that depending on the application, the ratio R ofthe grating period 34 of the diffraction grating 28 to the pixel pitch62 of the pixel array 38 along the grating axis 30 can take severalvalues. In some implementations, the ratio R can be equal to or greaterthan two (i.e., R≥2); or equal to a positive integer greater than one(i.e., R=(n+1), where n={1, 2, . . . }); or equal to an integer power oftwo (i.e., R=2n, where n={1, 2, . . . }); or the like. In someimplementations, it may be beneficial or required that the gratingperiod 34 be not only larger than, but also not too close to the pixelpitch 62 along the grating axis 30. For example, in someimplementations, it may be advantageous that the grating period 34 be atleast about twice the underlying pixel bank pitch 62 to allow for eachpair of adjacent pixel banks to sufficiently subsample the resultantmodulated diffracted wavefront 36, whose spatial modulation rate isdictated by the properties of the diffraction grating 28, near or atNyquist rate. This Nyquist, or nearly Nyquist, subsampling can allow forthe direct removal of the modulating component I_(mod) from the measuredsignal I by standard signal processing techniques. Once removed, themodulating signal I_(mod) may be manipulated independently of the basecomponent I_(base). In some implementations, undersampling effects canarise if the pixel pitch 62 along the grating axis 30 is notsufficiently smaller than the grating period 34. In such scenarios, itmay become useful or even necessary to alter the grating design toprovide two different sub-gratings with a sufficient relative phaseoffset between them to allow for signal subtraction.

For example, in the illustrated embodiment of FIG. 7, the ratio R of thegrating period 34 to the pixel pitch 62 along the grating axis 30 issubstantially equal to two. It will be understood that in such a case,adjacent pixel banks will sample complimentary spatial phases of theintensity profile 70 of the detected diffracted wavefront 36, that is,spatial parts of the intensity profile 70 that are phase-shifted by 180°relative to each other. This can be expressed mathematically as follows:|ϕ_(bank,n+1)−ϕ_(bank,n)|=π, where ϕ_(bank,n+1) and ϕ_(bank,n) are thespatial phases of the intensity profile 70 measured by the (n+1)^(th)and the n^(th) pixel banks of the pixel array 38, respectively. Such aconfiguration can allow for a direct deconvolution of the modulatingcomponent I_(mod) and the base component I_(base) through thesubsampling of the interference pattern resulting from the incident wavefronts interaction:

I _(base) =I(bank_(n))+I(bank_(n+1)),  (3)

I _(mod) =I(bank_(n))−I(bank_(n+1)).  (4)

Referring still to FIG. 7, in the illustrated embodiment, thediffraction grating 28 has a duty cycle of 50% (i.e., ridges 52 andgrooves 54 of equal width), and each light-sensitive pixel 40 ispositioned under and in vertical alignment with either a correspondingone of the ridges 52 or a corresponding one of the grooves 54. However,other arrangements can be used in other embodiments, non-limitingexamples of which are shown in FIGS. 8A to 8C. First, in FIG. 8A, thediffraction grating 28 has a duty cycle of 50%, but is laterally shiftedby a quarter of the grating period 34 compared to the embodiment FIG. 7.As a result, each light-sensitive pixel 40 is positioned under and invertical alignment with a transition 78 between a corresponding one ofthe ridges 52 and a corresponding adjacent one of the grooves 54.Second, in FIG. 8B, the diffraction grating 28 has a duty cycle of 50%,but compared to the embodiment of FIG. 7, the ratio R of the gratingperiod 34 to the pixel pitch 62 along the grating axis 30 is equal tofour rather than two. There are therefore two light-sensitive pixels 40under each of the ridges 52 and each of the grooves 54. Finally, in FIG.8C, the ratio R of the grating period 34 to the pixel pitch 62 along thegrating axis 30 is equal to two, as in FIG. 7, but the duty cycle of thediffracting grating is different from 50%.

In some implementations, for example in backside-illuminatedarchitectures with high chief-ray angle optical systems, the diffractiongrating may be designed to follow the designed chief-ray-angle offset ofthe microlens array relative to their light-sensitive pixel so that eachcorresponding chief ray will pass through the center of the intendedgrating feature and its subsequent microlens. Such a configuration canensure appropriate phase offsets for highly constrained optical systems.This means that, in some embodiments, the degree of vertical alignmentbetween the features of the diffraction grating (e.g., ridges andgrooves) and the underlying light-sensitive pixels can change as afunction of position within the pixel array, for example as one goesfrom the center to the edge of the pixel array, to accommodate apredetermined chief-ray-angle offset. For example, in some regions ofthe pixel array, each light-sensitive pixel may be positioned directlyunder a groove or a ridge of the diffraction grating, while in otherregions of the pixel array, each light-sensitive pixel may extend underboth a portion of a ridge and a portion of a groove.

In the implementations of FIGS. 7 and 8A to 8C, the diffraction grating28 is oriented with respect to the underlying pixel array 38 so that thegrating axis 30 is parallel to one of the two orthogonal pixel axes 58,60 (and thus perpendicular to each other). However, referring to FIGS.9A and 9B, there are illustrated two other possible embodiments in whichthe grating axis 30 is oblique to both the two orthogonal pixel axes 58,60. This is, in FIG. 9A, the grating axis 30 is oriented at an angleθ=45° with respect to each one of the pixel axes 58, 60, while in FIG.9B, the grating axis is oriented at angle θ≈26.565° with respect to thepixel axis 58. It is to be noted that in the oblique configurationsillustrated in FIGS. 9A and 9B, the pixel pitch 62 along the gratingaxis 30 remains smaller than the grating period. It is also to be notedthat pixel banks such as defined above, that is, groups of pixelsarranged along a line transverse to the grating axis 30 of the overlyingdiffraction grating 28 can also be defined in oblique configurations.For example, FIG. 9A includes a first group of pixels 40 ₁ that belongto a first pixel bank located under ridge 52, and a second group ofpixels 40 ₂ that belongs to a second pixel bank located an adjacentgroove 54.

Referring now to FIG. 10, there is shown a schematic partially explodedside view of an embodiment of a light field imaging device 20 suitablefor color imaging applications. The light field imaging device 20 sharessimilarities with the one shown in FIGS. 1 and 2, in that it includes adiffraction grating 28 disposed on top of a color filter array 42, whichis itself disposed on top of a pixel array 38 of light-sensitive pixels40. The diffraction grating 28 is a binary phase transmission gratinghaving a duty cycle of 50% and a periodic refractive index modulationpattern 32 consisting of alternating sets of ridges 52 and grooves 54.The color filter array 42 has a Bayer pattern, of which FIG. 10 depictsa row of alternating green (G) and blue (B) filters. FIG. 10 alsodepicts schematically the propagation of light through the device 20. Inoperation, the diffraction grating 28 receives and diffracts an opticalwavefront 26 originating from the scene 22 to generate a diffractedwavefront 36. For simplicity, it is assumed that the diffraction grating28 of FIG. 10 is achromatic in the spectral range encompassing green andblue light. The color filter array 42 receives and spatio-spectrallyfilters the diffracted wavefront 36 prior to its detection by theunderlying pixel array 38. The operation of the light field imagingdevice 20 is therefore based on a directly spatio-and-chromaticallysampled diffracted wavefront 36 enabled by the provision of a periodicdiffraction grating 28 deposed on top of a sensor structure including acolor filter array 42 and an underlying pixel array 38.

As in FIG. 7, the diffracted wavefront 36 produced by the diffractiongrating 28 in FIG. 10 defines a diffraction pattern characterized by aspatially varying intensity profile 70 along the grating axis 30. Also,the diffraction grating 28 and the pixel array 38 are disposed relativeto each other such that the light-receiving surface 68 of the pixelarray 38 is positioned in the near field of the diffraction grating 28,where the spatial period 74 of the intensity profile 70 of the detecteddiffracted wavefront 36 substantially matches the grating period 34 ofthe diffraction grating 28.

It will be understood that the intensity profile 70 of the diffractedwavefront 36 that is detected by the pixel array 38 afterspatio-spectral filtering by the color filter array 42 is a combinationor superposition of the portions of the diffracted wavefront 36 filteredby the red filters, the portions of the diffracted wavefront 36 filteredby the green filters, and the portions of the diffracted wavefront 36filtered by the blue filters. As such, using a standard RGB Bayerpattern as an example, the modulating component I_(mod) and the basecomponent I_(base) of the intensity profile I can be split into theirrespective color components as follows:

I _(R) ˜I _(mod,R)(depth info)×I _(base,R)(2D image),  (5)

I _(G) ˜I _(mod,G)(depth info)×I _(base,G)(2D image),  (6)

I _(B) ˜I _(mod,B)(depth info)×I _(base,B)(2D image).  (7)

In FIG. 10, the intensity profiles I_(G) and I_(B) are depicted indashed and dotted lines, respectively.

As in FIG. 7, the ratio R of the grating period 34 of the diffractiongrating 28 to the pixel pitch 62 of the pixel array 38 along the gratingaxis 30 is equal to two in the embodiment of FIG. 10, and therelationship |ϕ_(bank,n+1)−ϕ_(bank,n)|=ϕ introduced above applies. In astandard RGB Bayer pattern, the red and blue filters are always locatedin adjacent pixel banks in a Bayer pattern, the signals I_(R) and I_(B),which are associated with the sparsely sampled red and blue components,will be in antiphase relative to each other. Meanwhile, because greenfilters are present in all pixel banks, the signal I_(G), which isassociated with the densely sampled green components, will contain bothin-phase and out-of-phase contributions.

In the implementations described so far, the diffraction gratingassembly was depicted as including only one diffracting grating.However, referring to FIG. 11, in other implementations, the diffractiongrating assembly 24 includes a plurality of diffracting gratings 28 a,28 b, where the diffracting gratings 28 a, 28 b are arranged in atwo-dimensional grating array disposed over the color filter array 42.In FIG. 11, the diffracting grating assembly 24 includes sixteendiffraction gratings, but this number is provided for illustrativepurposes and could be varied in other embodiments. For example,depending on the application, the number of diffraction gratings 28 a,28 b in the diffraction grating assembly 24 can range from one to up tomillions (e.g., a 20-megapixel pixel array 38 could have up to 2.8million diffraction gratings on top of it). It is to be noted that otherthan their grating axis orientation, every diffraction grating 28 of thediffraction grating assembly 24 depicted in FIG. 11 is a binary phasegrating including alternating sets of parallel ridges 52 and grooves 54having the same duty cycle of 50%, the same grating period 34, and thesame number of repetitions of the grating period 34, although in otherembodiments each of these parameters can be varied from one diffractiongrating 28 to the another. More particularly, each one of thediffraction gratings 28 in FIG. 11 includes two repetitions of thegrating period 34. However, it will be understood that this number canbe varied depending on the application, for example between two and tenrepetitions in some embodiments.

In some implementations, the plurality of diffraction gratings 28includes multiple sets 80 a, 80 b of diffraction gratings 28, where thegrating axes 30 a, 30 b of the diffraction gratings 28 of different onesof the sets 80 a, 80 b have different orientations. For example, in FIG.11, the multiple sets 80 a, 80 b consist of a first set 80 a ofdiffraction gratings 28 and a second set 80 b of diffraction gratings28, the grating axes 30 a of the diffraction gratings 28 of the firstset 80 a extending substantially perpendicularly to the grating axes 30b of the diffraction gratings 28 of the second set 80 b. The firstgrating axes 30 a are parallel to the first pixel axis 58, while thesecond grating axes 30 b are parallel to the second pixel axis 60. Inthe illustrated embodiment, the diffraction gratings 28 of the first set80 a and second set 80 b are arranged to alternate in both rows andcolumns, resulting in a checkerboard pattern. Of course, any othersuitable regular or irregular arrangement, pattern or mosaic oforthogonally oriented gratings can be envisioned in other embodiments.For example, the orthogonally oriented gratings could be arranged toalternate only in rows or only in columns or arranged randomly.Furthermore, other embodiments can include more than two sets ofdiffraction gratings, which may or may not be orthogonal with respect toone another. For example, in some implementations, the diffractiongrating assembly can include up to 24 different sets of diffractiongratings.

It will be understood that providing a diffraction grating assembly withdiffracting gratings having different grating axis orientations can beadvantageous or required in some implementations since diffractionoccurs along the grating axis of an individual diffraction grating. Thismeans that when only a single grating orientation is present in thediffraction grating assembly, light coming from objects of the scenethat extend perpendicularly to this single grating orientation will notbe diffracted. In some implementations, providing two sets oforthogonally oriented gratings (e.g., horizontally and verticallyoriented gratings) can be sufficient to capture sufficient light fieldimage data about the scene. The concept of using diffraction gratingassemblies with two or more grating orientations can be taken to thelimit of completely circular diffraction gratings having increasingperiodicity radially form the center, which would provide a near perfectFourier plane imager.

Referring to FIGS. 12A to 12C, there are illustrated other examples ofgrating arrangements in diffraction grating assemblies including aplurality of diffraction gratings. In FIG. 12A, the diffraction gratingassembly 24 includes two sets 80 a, 80 b of orthogonally orienteddiffraction gratings 28 that alternate only in columns. The grating axisorientation of one set 80 a is along one pixel axis 58, and the gratingaxis orientation of the other set 80 b is along the other pixel axis 60.In FIG. 12B, the diffraction grating assembly 24 includes four sets 80 ato 80 d of diffraction gratings 28 whose grating axes 34 a to 34 d areoriented at 0°, 33°, 66° and 90° with respect to the horizontal pixelaxis 58. In FIG. 12C, the diffraction grating assembly 24 includes foursets 80 a to 80 d of diffraction gratings 28 whose grating axes 34 a to34 d are oriented at 0°, 45°, 90° and −45° with respect to thehorizontal pixel axis 58. It will be understood that in each of FIGS.12A to 12C, the depicted diffraction gratings 28 can represent a unitcell of the diffraction grating assembly 24, which is repeated aplurality of times.

Referring now to FIG. 13, there is shown an embodiment of a light fieldimaging device 20 that is suitable for color-based applications, butdoes not include a color filter array disposed between the diffractiongrating assembly 24 and underlying pixel array 38. Rather, in theillustrated embodiment, the diffraction grating assembly 24 includes anarray of diffraction gratings 28, each of which includes a gratingsubstrate 46 having a refractive index modulation pattern 32 formedthereon (e.g., made of alternating series of ridges 52 and grooves 54).The grating substrate 46 of each diffraction grating 28 also includes aspectral filter material or region 82 configured to spectrally filterthe diffracted wavefront 36 prior to its detection by the plurality oflight-sensitive pixels 40. In some implementations, each one of thediffraction grating 28 can be made of a material tailored to filter adesired spectral component, for example by incorporating a suitable dyedopant in the grating substrate 46.

Referring still to FIG. 13, the plurality of diffraction gratings 28 ofthe diffraction grating assembly 24 thus forms a color filter array inwhich each color filter is embodied by a corresponding one of thediffraction gratings 28. In other words, each one of the diffractiongratings 28 can be individually designed and tailored so that it formsto its own respective color filter in the color filter array. In FIG.13, the color filter array formed by the plurality of diffractiongratings 28 is arranged in a Bayer pattern, so that the gratingsubstrate 46 of each diffraction grating 28 acts as a red pass filter, agreen pass filter or a blue pass filter. Of course, the color filterarray defined by the plurality of diffraction gratings 28 can beoperated outside the visible region of the electromagnetic spectrum andits mosaic color pattern is not limited to Bayer-type patterns, but canbe applied to any appropriate mosaic color pattern, including thoselisted above.

In some implementations, the light field imaging device can includewavefront conditioning optics in front of the diffraction grating. Thewavefront conditioning optics can be configured to collect, direct,transmit, reflect, refract, disperse, diffract, collimate, focus orotherwise act on the optical wavefront incident from the scene prior toit reaching the diffraction grating assembly. The wavefront conditioningoptics can include lenses, mirrors, filters, optical fibers, and anyother suitable reflective, refractive and/or diffractive opticalcomponents, and the like. In some implementations, the wavefrontconditioning optics can include focusing optics positioned andconfigured to modify the incident wavefront in such a manner that it maybe sampled by the light field imaging device.

Referring now to FIG. 14, another possible embodiment of a light fieldimaging device 20 is illustrated and includes dispersive optics 84disposed in a light path of the optical wavefront 26 between the sceneand the diffraction grating assembly. The dispersive optics 84 isconfigured to receive and disperse the incoming optical wavefront 26.The dispersive optics 84 can be embodied by any optical component orcombination of optical components in which electromagnetic beams aresubject to spatial spreading as a function of wavelength as they passtherethrough (e.g., by chromatic aberration). In the embodiment of FIG.14, the dispersive optics 84 is a focusing lens, for simplicity.However, it will be understood that, in other embodiments, thedispersive optics 84 can be provided as an optical stack including alarger number of optical components (e.g., focusing and defocusingoptics) that together act to disperse the optical wavefront 26 before itimpinges on the diffraction grating assembly 24 (e.g., due to theirintrinsic chromatic aberration).

For exemplary purposes, it is assumed in FIG. 14 that the opticalwavefront 26 originating from the scene 22 is a superposition of wavescontaining multiple wavelengths of light, for example a green component(dashed line) and a blue component (dotted line). Each color componentsof the optical wavefront 26, by the nature of its energy-dependentinteraction with the dispersive optics 84, will follow a slightlydifferent optical path, leading to a chromatic dependence in thephase-shift introduced by the diffraction grating 28. In other words,the chromatic spread of the optical wavefront 26, as sampled through theangle-dependent diffraction produced by the diffractive grating 28, canprovide coarse depth information about the optical wavefront 26. In suchscenarios, the finer details of the depth information can be obtainedfrom a comparative analysis of the modulating components I_(mod,R) andI_(mod,B), which are phase-shifted relative to each other due to theiroptical path differences, as sampled by the color filter array 42.

It is to be noted that in the case of monochromatic plane opticalwavefront impinging on a focusing lens such as shown in FIG. 14, thefocusing lens gradually refracts and focuses the wavefront as ittraverses the lens. It will be understood that the cross-sectional areaof the wavefront reaching the diffraction grating assembly will belarger if the diffraction grating assembly is located out (either beforeor after) of the focal plane of the focusing lens that if it is locatedin the focal plane. Accordingly, the diffracted wavefront will besampled by a greater number of light-sensitive pixels in theout-of-focus than in the in-focus configuration.

Referring to FIGS. 15 and 16, in some implementations, the light fieldimaging device 20 can include pixel array circuitry 86 disposed eitherbetween the diffraction grating assembly and the pixel array, in afrontside illumination configuration (FIG. 15), or under the pixel array38, in a backside illumination configuration (FIG. 16). Moreparticularly, the diffraction grating assembly 24 can be directly etchedinto overlying silicon layers in the case of a frontside illuminationarchitecture (FIG. 15), or placed directly atop the microlens array 64and the color filter array 42 in the case of a backside illuminationarchitecture (FIG. 16). In frontside illumination technology, the pixelarray circuitry 86 includes an array of metal wiring (e.g., a siliconlayer hosting a plurality of metal interconnect layers) connecting thecolor filters 44 to their corresponding light-sensitive pixels 40.Meanwhile, backside illumination technology provides opportunities fordirectly sampling the diffracted wavefront 36 produced by diffraction ofan optical waveform 26 by the diffraction grating assembly 24. As lightdoes not have to pass through the array of metal wiring of the pixelarray circuitry 86 before reaching the pixel array 38, which otherwisewould result in a loss of light, more aggressive diffraction gratingdesigns with increased periodicity can be implemented. Also, the shorteroptical stack configuration, as shown in FIG. 16, can allow for thediffraction grating assembly 24 to be positioned in much closerproximity to the light-receiving surface 68 of the pixel array 38,thereby decreasing the risk of higher-order diffractive effects whichcould cause undesirable cross-talk between pixel banks. Similarly, thedecreased pixel size can allow for direct subsampling of the diffractiongrating by the existing imaging wells.

Referring now more specifically to FIG. 16, there is shown abackside-illuminated light field imaging device 20 for capturing lightfield image data about a scene 22. The device 20 includes a substrate 88having a front surface 90 and a back surface 92; a diffraction gratingassembly 24 disposed over the back surface 92 of the substrate 88 andconfigured to receive an optical wavefront 26 originating from the scene22; a pixel array 38 formed in the substrate 88; and pixel arraycircuitry 86 disposed under the front surface 90 and coupled to thepixel array 38. The diffraction grating assembly 24 includes at leastone diffraction grating 28 having a grating axis 30 and a refractiveindex modulation pattern 32 having a grating period 34 along the gratingaxis 30. The diffraction grating 28 diffracts the optical wavefront 26to generate a diffracted wavefront 36. The pixel array 38 includes aplurality of light-sensitive pixels 40 configured to receive, throughthe back surface 92, and detect, as the light field image data, thediffracted wavefront 36. As mentioned above, the pixel array 38 has apixel pitch 62 along the grating axis 30 that is smaller than thegrating period 34. As mentioned above, an advantage of backsideillumination sensor technology in the context of the present techniquesis that the diffraction grating assembly 24 can be positioned closer tothe light-receiving surface 68 of the pixel array 38 than in frontsideillumination applications. For example, in some backside illuminationimplementations, a separation distance 72 between the refractive indexmodulation pattern 32 of the diffraction grating 28 and thelight-receiving surface 68 of the pixel array 38 can range from about0.5 μm to about 5 μm, for example between 1 and 3 μm.

In color imaging applications, the backside-illuminated light fieldimaging device 20 can include a color filter array 42 disposed over theback surface 92 and including a plurality of color filters 44 arrangedin a mosaic color pattern, for example a Bayer pattern. The color filterarray 42 spatially and spectrally filters the diffracted wavefront 36according to the mosaic color pattern prior to its detection by theplurality of light-sensitive pixels 40. The device 20 also includes amicrolens array 64 disposed over the color filter array 42 and includinga plurality of microlenses 66, each of which is optically coupled to acorresponding one of the plurality of the color filters 44. In FIG. 16,the diffraction grating 28 also includes a grating substrate 46including a top surface 48 having the refractive index modulationpattern 32 formed thereon and a bottom surface 50 disposed over themicrolens array 64. It is to be noted that the diffraction gratingassembly 24, the pixel array 38, the color filter array 42 and themicrolens array 64 of the backside-illuminated light field imagingdevice 20 can share similar features to those described above.

It is to be noted that backside illuminated and stacked-architecturedevices are often employed in situations where sensor footprint is anissue (e.g., smartphone modules, tablets, webcams) and are becomingincreasingly complex in design. In some implementations, the presenttechniques involve positioning a diffraction grating assembly directlyon top of an existing sensor architecture as an independent process.Therefore, using the present techniques with backside illuminationsensor technology can represent a flexible opportunity for sensor-leveldepth sensing optics, as it does not require a complete sensor orpackaging redesign as is the case for microlens or coded apertureapproaches. Furthermore, the modest z-stack increase of the order ofmicrometers resulting from the integration of the diffraction gratingassembly on top of the sensor can similarly simplify packagingrequirements and implementation in the overall optical stack of thesensor module. Additionally, the backside illumination manufacturingprocess itself does not require a direct etch into existing siliconlayers as would be the case in frontside illumination technology. It isto be noted that for backside-illuminated devices with larger pixelpitch values and certain frontside illuminated devices, the diffractiongrating assembly itself can act as a color filter array (see, e.g., FIG.13), which can reduce the manufacturing complexity and/or the overallheight of the optical stack. It is also to be noted that the differentlayers of the light field imaging device may be stacked and spaced-apartaccording to geometrical parameters supporting the desired opticalfunctionalities.

Diffraction Grating Assembly Implementations

Referring to FIG. 17, in accordance with another aspect, the presentdescription also relates to a diffraction grating assembly 24 for usewith an image sensor 94 including a pixel array 38 having a plurality oflight-sensitive pixels 40 to capture light field image data about ascene 22. The diffraction grating assembly 24, which is configured to bedisposed over the pixel array 38, can share many similarities with thosedescribed above in the context of light field imaging deviceimplementations, insofar as it includes a diffraction grating 28 havinga grating axis 30 and a refractive index modulation pattern 32 having agrating period 34 along the grating axis 30, the grating period 34 beinglarger than a pixel pitch 62 of the pixel array 38 along the gratingaxis 30. For example, a ratio of the grating period 34 to the pixelpitch 62 along the grating axis 30 can be equal to two or an integermultiple of two. In some implementations, the diffraction grating 28 canbe a binary phase grating and the refractive index modulation pattern 32can include alternating ridges 52 and grooves 54. The diffractiongrating 28 is configured to receive and diffract an optical wavefront 26originating from the scene 22 to generate a diffracted wavefront 36 fordetection by the light-sensitive pixels 40 as the light field imagedata. In some implementations intended for color imaging applications,the diffraction grating assembly 24 is configured to be disposed over acolor filter array 42 of the image sensor 94. The color filter array 42is disposed over pixel array 38 and configured to spatially andspectrally filter the diffracted wavefront 36 prior to its detection bythe plurality of light-sensitive pixels 40.

Depending on the application, the diffraction grating assembly 24 caninclude a single diffraction grating 28 or a plurality of diffractiongratings 28 arranged in a two-dimensional grating array disposed overthe pixel array 38.

Method Implementations

In accordance with another aspect, the present description also relatesto various light field imaging methods, including a method of capturinglight field image data about a scene as well as a method of providing 3Dimaging capabilities to a conventional 2D image sensor. These methodscan be performed with light field imaging devices and diffractiongrating assemblies such as those described above, or with other similardevices and assemblies.

Referring to FIG. 18, there is provided a flow diagram of an embodimentof a method 200 of capturing light field image data about a scene. Themethod includes a step 202 of diffracting an optical wavefrontoriginating from the scene with a diffraction grating. The diffractiongrating has a grating axis and a grating period along the grating axis.The diffraction grating is configured to diffract the incident opticalwavefront to generate a diffracted wavefront. The diffracted wavefrontcan be characterized by an intensity profile along the grating axis. Insome implementations, the diffracting step 202 can include diffractingthe optical wavefront in a waveband ranging from 400 nm (blue end ofvisible spectrum) to 1550 nm (telecommunication wavelengths), forexample from 400 nm to 650 nm. In some implementations, the diffractiongrating is one of a plurality of diffraction gratings that together forma diffraction grating assembly. In such implementations, the method 200of FIG. 18 can be performed simultaneously for each diffraction gratingof the diffraction grating assembly.

In some implementations, the method 200 can include a step of providingthe diffraction grating as a phase grating, for example a binary phasegrating. The binary phase grating can include alternating ridges andgrooves periodically spaced-apart at the grating period. The method 200can include a step of selecting the grating period in a range between 1μm to 20 μm. The method 200 can also include a step of setting a stepheight of the ridges relative to the grooves to control an optical pathdifference between adjacent ridges and grooves. For example, in someimplementations, the step height can be set to provide, at a givenwavelength of the optical wavefront, a half-wave optical path differencebetween the ridges and the grooves. Of course, other values of opticalpath difference can be used in other implementations.

Referring still to FIG. 18, the method 200 also includes a step 204 ofspatio-spectrally filtering the diffracted wavefront with a color filterarray to produce a filtered wavefront. It is to be noted that this step204 is optional and can be omitted in some implementations, for examplein monochrome imaging applications.

The method 200 can further include a step 206 of detecting thespatio-spectrally filtered wavefront as the light field image data. Thedetecting step 206 can be performed with a pixel array comprising aplurality of light-sensitive pixels disposed under the color filterarray. However, when the spatio-spectral filtering step 204 is omitted,there is no color filter array disposed between the diffraction gratingassembly and the pixel array, and the detecting step 206 involves thedirect detection of the diffracted wavefront with the plurality oflight-sensitive pixels. As mentioned above with respect to deviceimplementations, the grating period of the diffraction grating isselected to be larger than the pixel pitch of the pixel array along thegrating axis. As also mentioned above, the separation distance betweenthe top surface of the diffraction grating (i.e., the refractive indexmodulation pattern) and the light-receiving surface of the underlyingpixel array is selected so that the filtered or diffracted wavefront isdetected in a near-field diffraction regime, where the intensity profileof the diffracted wavefront along the grating axis has a spatial periodthat substantially matches the grating period. For example, in someimplementations, the method can include a step of setting the separationdistance to a value that is less than about ten times a centerwavelength of the optical wavefront to detect the filtered or diffractedwavefront in the near field.

In some implementations, the diffraction grating can be provided with aduty cycle of about 50%, and the method 200 can include a step ofpositioning each light-sensitive pixel under and in alignment witheither a ridge or a groove of the diffraction grating, or under and inalignment with a transition or boundary between a ridge and an adjacentgroove. In some implementations, the method 200 can include a step ofsetting a ratio of the grating period to the pixel pitch along thegrating axis to be substantially equal to two or an integer multiple oftwo.

Referring still to FIG. 18, in some implementations, the plurality oflight-sensitive pixels can be arranged in a rectangular pixel griddefined by two orthogonal pixel axes, and the method 200 can include astep of orienting the grating axis either parallel to one of the twoorthogonal pixel axes or oblique to both the two orthogonal pixel axes.For example, in some orthogonal implementations, one half of thediffraction gratings can be oriented along one pixel axis, and the otherhalf can be oriented along the other pixel axis. One possible obliqueconfiguration can include orienting the diffraction gratings at an angleof 45° with respect to each pixel axis.

In some implementations, the method 200 can further include an optionalstep of spectrally dispersing the optical wavefront prior to diffractingthe optical wavefront.

Referring now to FIG. 19, there is provided a flow diagram of a method300 of providing 3D imaging capabilities, for example depth mappingcapabilities, to an image sensor viewing a scene and including a pixelarray having a plurality of light-sensitive pixels. For example, theimage sensor can be a conventional or custom-designed frontside- orbackside illuminated CMOS or CCD sensor.

The method 300 includes a step 302 disposing a diffraction gratingassembly in front of the image sensor. The diffraction grating assemblyincludes at least one diffraction grating, each of which having agrating axis and a grating period along the grating axis. The gratingperiod is selected to be larger than a pixel pitch of the pixel arrayalong the grating axis. For example, in some implementations, thegrating period can be larger than the pixel pitch along the grating axisby a factor of two or more. In some implementations, the disposing step302 can include positioning the diffraction grating assembly at aseparation distance from the pixel array which is selected such that anoptical path length of the diffracted wavefront prior to detection bythe light-sensitive pixels is less than about ten times a centerwavelength of the optical wavefront. Such a configuration allowsdetection of the diffracted wavefront in a near-field diffractionregime. In some implementations, the disposing step 320 can includeorienting the grating axis either parallel to one of two orthogonalpixel axes of the pixel array or oblique (e.g., at 45°) to the pixelaxes.

In some implementations, the method 300 can include a step of providingthe diffraction grating as a phase grating, for example a binary phasegrating. The binary phase grating can include a series of ridgesperiodically spaced-apart at the grating period, interleaved with aseries of grooves also periodically spaced-apart at the grating period.The method 300 can include a step of selecting the grating periodbetween 1 μm to 20 μm. The method 300 can also include a step of settinga step height of the ridges relative to the grooves to control anoptical path difference between adjacent ridges and grooves. Asmentioned above, the step height can be selected to provide apredetermined optical path difference between the ridges and thegrooves. In some implementations, the diffraction grating can beprovided with a duty cycle of about 50% and the diffraction gratingassembly can be positioned over the pixel array such that each ridge andeach groove extends over and in alignment with a corresponding one ofthe light-sensitive pixels, or alternatively such that each transitionor junction between adjacent ridges and grooves extends over and inalignment with a corresponding one of the light-sensitive pixels.

Referring still to FIG. 19, the method 300 also includes a step 304 ofreceiving and diffracting an optical wavefront originating from thescene with the diffraction grating to generate a diffracted wavefront,and a step 306 of detecting the diffracted wavefront with thelight-sensitive pixels. In color imaging applications, the method 300can include an optional step 308 of spatio-spectrally filtering thediffracted wavefront with a color filter array prior to the detectingstep 306. In some implementations, the method 300 can further include anoptional step of spectrally dispersing the optical wavefront prior todiffracting the optical wavefront.

Of course, numerous modifications could be made to the embodimentsdescribed above without departing from the scope of the presentdescription.

1. A light field imaging device for capturing light field image dataabout a scene, the light field imaging device comprising: a diffractiongrating assembly configured to receive an optical wavefront originatingfrom the scene, the diffraction grating assembly comprising adiffraction grating having a grating axis and a refractive indexmodulation pattern having a grating period along the grating axis, thediffraction grating diffracting the optical wavefront to generate adiffracted wavefront; and a pixel array comprising a plurality oflight-sensitive pixels disposed under the diffraction grating assemblyand detecting the diffracted wavefront as the light field image data,the pixel array having a pixel pitch along the grating axis that issmaller than the grating period.
 2. The light field imaging device ofclaim 1, further comprising a color filter array disposed over the pixelarray and comprising a plurality of color filters arranged in a mosaiccolor pattern, the color filter array spatially and spectrally filteringthe diffracted wavefront according to the mosaic color pattern prior todetection of the diffracted wavefront by the plurality oflight-sensitive pixels.
 3. The light field imaging device of claim 2,wherein each color filter is optically coupled to a corresponding one ofthe light-sensitive pixels.
 4. The light field imaging device of claim2, wherein each color filter is optically coupled to at least twocorresponding ones of the plurality of light-sensitive pixels.
 5. Thelight field imaging device of claim 2, wherein each color filter is oneof a red pass filter, a green pass filter and a blue pass filter.
 6. Thelight field imaging device of claim 2, wherein the mosaic color patternis a Bayer pattern.
 7. The light field imaging device of claim 1,wherein the diffraction grating is configured to diffract the opticalwavefront in a waveband ranging from 400 nanometers to 1550 nanometers.8. The light field imaging device of claim 1, wherein the grating periodranges from 1 micrometer to 20 micrometers.
 9. The light field imagingdevice of claim 1, wherein the diffraction grating includes between twoand ten repetitions of the grating period.
 10. The light field imagingdevice of claim 1, wherein the diffraction grating is a phase grating.11. The light field imaging device of claim 10, wherein the diffractiongrating is a binary phase grating and the refractive index modulationpattern comprises a series of ridges periodically spaced-apart at thegrating period, interleaved with a series of grooves periodicallyspaced-apart at the grating period.
 12. The light field imaging deviceof claim 11, wherein the diffraction grating has a duty cycle of about50%.
 13. The light field imaging device of claim 12, wherein eachlight-sensitive pixel is positioned under and in alignment with either acorresponding one of the ridges or a corresponding one of the grooves.14. The light field imaging device of claim 12, wherein eachlight-sensitive pixel is positioned under and in alignment with atransition between a corresponding one of the ridges and a correspondingadjacent one of the grooves.
 15. The light field imaging device of claim11, wherein the diffraction grating has a duty cycle different from 50%.16. The light field imaging device of claim 11, wherein the series ofridges has a step height with respect to the series of grooves, the stepheight ranging from 0.2 micrometer to 1 micrometer.
 17. The light fieldimaging device of claim 1, wherein a separation distance between therefractive index modulation pattern of the diffraction grating and alight-receiving surface of the pixel array ranges from 0.5 micrometer to20 micrometers.
 18. The light field imaging device of claim 1, wherein aseparation distance between the refractive index modulation pattern ofthe diffraction grating and a light-receiving surface of the pixel arrayis less than about ten times a center wavelength of the opticalwavefront.
 19. The light field imaging device of claim 1, wherein aratio of the grating period to the pixel pitch along the grating axis issubstantially equal to two.
 20. The light field imaging device of claim1, wherein the plurality of light-sensitive pixels is arranged in arectangular pixel grid defined by two orthogonal pixel axes, and whereinthe grating axis is either parallel to one of the two orthogonal pixelaxes or oblique to both the two orthogonal pixel axes.
 21. The lightfield imaging device of claim 1, wherein the pixel pitch ranges from 1micrometer to 10 micrometers.
 22. The light field imaging device ofclaim 1, further comprising dispersive optics disposed in a light pathof the optical wavefront between the scene and the diffraction gratingassembly, the dispersive optics being configured to receive andspectrally disperse the optical wavefront.
 23. The light field imagingdevice of claim 1, further comprising a microlens array disposed overthe pixel array and comprising a plurality of microlenses, eachmicrolens being optically coupled to a corresponding one of thelight-sensitive pixels.
 24. The light field imaging device of claim 1,further comprising pixel array circuitry disposed either under the pixelarray, in a backside illumination configuration, or between thediffraction grating assembly and the pixel array, in a frontsideillumination configuration.
 25. The light field imaging device of claim1, wherein the diffraction grating is one of a plurality of diffractiongratings of the diffraction grating assembly, the plurality ofdiffraction gratings being arranged in a two-dimensional grating arraydisposed over the pixel array.
 26. The light field imaging device ofclaim 25, wherein the plurality of diffraction gratings comprisesmultiple sets of diffraction gratings, the grating axes of thediffraction gratings of different ones of the sets having differentorientations.
 27. The light field imaging device of claim 26, whereinthe multiple sets of diffraction gratings comprise a first set ofdiffraction gratings and a second set of diffraction gratings, thegrating axes of the diffraction gratings of the first set extendingsubstantially perpendicularly to the grating axes of the diffractiongratings of the second set.
 28. The light field imaging device of claim24, wherein each diffraction grating comprises a grating substrateincluding a top surface having the refractive index modulation patternformed thereon, the grating substrate comprising a spectral filtermaterial or region configured to spectrally filter the diffractedwavefront prior to detection of the diffracted wavefront by theplurality of light-sensitive pixels, the plurality of diffractiongratings thus forming a color filter array.
 29. The light field imagingdevice of claim 28, wherein the grating substrate of each diffractiongrating acts as one of a red pass filter, a green pass filter and a bluepass filter.
 30. The light field imaging device of claim 28, wherein thecolor filter array is arranged in a Bayer pattern.
 31. Abackside-illuminated light field imaging device for capturing lightfield image data about a scene, the backside-illuminated light fieldimaging device comprising: a substrate having a front surface and a backsurface; a diffraction grating assembly disposed over the back surfaceof the substrate and configured to receive an optical wavefrontoriginating from the scene, the diffraction grating assembly comprisinga diffraction grating having a grating axis and a refractive indexmodulation pattern having a grating period along the grating axis, thediffraction grating diffracting the optical wavefront to generate adiffracted wavefront; a pixel array formed in the substrate andcomprising a plurality of light-sensitive pixels configured to receivethrough the back surface and detect as the light field image data thediffracted wavefront, the pixel array having a pixel pitch along thegrating axis that is smaller than the grating period; and pixel arraycircuitry disposed under the front surface and coupled to the pixelarray.
 32. The backside-illuminated light field imaging device of claim31, further comprising a color filter array disposed over the backsurface and comprising a plurality of color filters arranged in a mosaiccolor pattern, the color filter array spatially and spectrally filteringthe diffracted wavefront according to the mosaic color pattern prior todetection of the diffracted wavefront by the plurality oflight-sensitive pixels.
 33. The backside-illuminated light field imagingdevice of claim 32, wherein the mosaic color pattern is a Bayer pattern.34. The backside-illuminated light field imaging device of claim 31,wherein the diffraction grating is a binary phase grating and therefractive index modulation pattern comprises a series of ridgesperiodically spaced-apart at the grating period, interleaved with aseries of grooves periodically spaced-apart at the grating period. 35.The backside-illuminated light field imaging device of claim 34, whereinthe diffraction grating has a duty cycle of about 50% and eachlight-sensitive pixel is positioned under and in alignment with either acorresponding one of the ridges or a corresponding one of the grooves.36. The backside-illuminated light field imaging device of claim 34,wherein the diffraction grating has a duty cycle of about 50% and eachlight-sensitive pixel is positioned under and in alignment with atransition between a corresponding one of the ridges and a correspondingadjacent one of the grooves.
 37. The backside-illuminated light fieldimaging device of claim 31, wherein a separation distance between therefractive index modulation pattern of the diffraction grating and alight-receiving surface of the pixel array ranges from 0.5 micrometer to5 micrometers.
 38. The backside-illuminated light field imaging deviceof claim 31, wherein a ratio of the grating period to the pixel pitchalong the grating axis is substantially equal to two.
 39. Thebackside-illuminated light field imaging device of claim 31, wherein theplurality of light-sensitive pixels is arranged in a rectangular pixelgrid defined by two orthogonal pixel axes, and wherein the grating axisis either parallel to one of the two orthogonal pixel axes or oblique toboth the two orthogonal pixel axes.
 40. The backside-illuminated lightfield imaging device of claim 31, wherein the pixel pitch ranges from 1micrometer to 5 micrometers.
 41. The backside-illuminated light fieldimaging device of claim 31, further comprising dispersive opticsdisposed in a light path of the optical wavefront between the scene andthe diffraction grating assembly, the dispersive optics being configuredto receive and spectrally disperse the optical wavefront.
 42. Thebackside-illuminated light field imaging device of claim 31, furthercomprising a microlens array disposed over the pixel array andcomprising a plurality of microlenses, each microlens being opticallycoupled to a corresponding one of the light-sensitive pixels.
 43. Thebackside-illuminated light field imaging device of claim 31, wherein thediffraction grating is one of a plurality of diffraction gratings of thediffraction grating assembly, the plurality of diffraction gratingsbeing arranged in a two-dimensional grating array disposed over thepixel array.
 44. The backside-illuminated light field imaging device ofclaim 43, wherein the plurality of diffraction gratings comprisesmultiple sets of diffraction gratings, the grating axes of thediffraction gratings of different ones of the sets having differentorientations.
 45. The backside-illuminated light field imaging device ofclaim 44, wherein the multiple sets of diffraction gratings comprise afirst set of diffraction gratings and a second set of diffractiongratings, the grating axes of the diffraction gratings of the first setextending substantially perpendicularly to the grating axes of thediffraction gratings of the second set.
 46. The backside-illuminatedlight field imaging device of claim 31, further comprising: a colorfilter array disposed over the back surface and comprising a pluralityof color filters, each of which optically coupled to a corresponding oneof the plurality of light-sensitive pixels, the color filter arrayspatially and spectrally filtering the diffracted wavefront prior todetection of the diffracted wavefront by the plurality oflight-sensitive pixels; and a microlens array disposed over the colorfilter array and comprising a plurality of microlenses, each microlensbeing optically coupled to a corresponding one of the plurality of thecolor filters, wherein the diffraction grating further comprises agrating substrate including a top surface having the refractive indexmodulation pattern formed thereon and a bottom surface disposed over themicrolens array.
 47. A light field imaging device comprising: adiffraction grating assembly comprising a diffraction grating having agrating axis and a refractive index modulation pattern having a gratingperiod along the grating axis; and a pixel array comprising a pluralityof light-sensitive pixels disposed under the diffraction grating, thepixel array having a pixel pitch along the grating axis that is smallerthan the grating period.
 48. The light field imaging device of claim 47,further comprising a color filter array disposed over the pixel arrayand comprising a plurality of color filters arranged in a mosaic colorpattern, the color filter array spatially and spectrally filtering thediffracted wavefront according to the mosaic color pattern prior todetection of the diffracted wavefront by the plurality oflight-sensitive pixels.
 49. The light field imaging device of claim 47,wherein the grating period ranges from 1 micrometer to 20 micrometers.50. The light field imaging device of claim 47, wherein the diffractiongrating is a binary phase grating and the refractive index modulationpattern comprises a series of ridges periodically spaced-apart at thegrating period, interleaved with a series of grooves periodicallyspaced-apart at the grating period.
 51. The light field imaging deviceof claim 50, wherein the diffraction grating has a duty cycle of about50% and each light-sensitive pixel is positioned under and in alignmentwith either a corresponding one of the ridges or a corresponding one ofthe grooves.
 52. The light field imaging device of claim 50, wherein thediffraction grating has a duty cycle of about 50% and eachlight-sensitive pixel is positioned under and in alignment with atransition between a corresponding one of the ridges and a correspondingadjacent one of the grooves.
 53. The light field imaging device of claim47, wherein a ratio of the grating period to the pixel pitch along thegrating axis is substantially equal to two.
 54. The light field imagingdevice of claim 47, wherein the plurality of light-sensitive pixels isarranged in a rectangular pixel grid defined by two orthogonal pixelaxes, and wherein the grating axis is either parallel to one of the twoorthogonal pixel axes or oblique to both the two orthogonal pixel axes.55. The light field imaging device of claim 47, further comprisingdispersive optics disposed in a light path of the optical wavefrontbetween the scene and the diffraction grating assembly, the dispersiveoptics being configured to receive and spectrally disperse the opticalwavefront.
 56. The light field imaging device of claim 47, wherein thediffraction grating is one of a plurality of diffraction gratings of thediffraction grating assembly, the plurality of diffraction gratingsbeing arranged in a two-dimensional grating array disposed over thepixel array.
 57. The light field imaging device of claim 56, wherein theplurality of diffraction gratings comprises multiple sets of diffractiongratings, the grating axes of the diffraction gratings of different onesof the sets having different orientations.
 58. The light field imagingdevice of claim 57, wherein the multiple sets of diffraction gratingscomprise a first set of diffraction gratings and a second set ofdiffraction gratings, the grating axes of the diffraction gratings ofthe first set extending substantially perpendicularly to the gratingaxes of the diffraction gratings of the second set.
 59. A diffractiongrating assembly for use with an image sensor comprising a pixel arrayhaving a plurality of light-sensitive pixels to capture light fieldimage data about a scene, the diffraction grating assembly comprising adiffraction grating having a grating axis and a refractive indexmodulation pattern having a grating period along the grating axis, thegrating period being larger than a pixel pitch of the pixel array alongthe grating axis, the diffraction grating being configured to receiveand diffract an optical wavefront originating from the scene to generatea diffracted wavefront for detection by the light-sensitive pixels asthe light field image data, the diffraction grating assembly beingconfigured to be disposed over the pixel array.
 60. The diffractiongrating assembly of claim 59, wherein the diffraction grating assemblyis configured to be disposed over a color filter array of the imagesensor, the color filter array being disposed over pixel array andconfigured to spatially and spectrally filter the diffracted wavefrontprior to detection of the diffracted wavefront by the plurality oflight-sensitive pixels.
 61. The diffraction grating assembly of claim59, wherein the diffraction grating is configured to diffract theoptical wavefront in a waveband ranging from 400 nanometers to 1550nanometers.
 62. The diffraction grating assembly of claim 59, whereinthe grating period ranges from 1 micrometer to 20 micrometers.
 63. Thediffraction grating assembly of claim 59, wherein the diffractiongrating is a binary phase grating and the refractive index modulationpattern comprises a series of ridges periodically spaced-apart at thegrating period, interleaved with a series of grooves periodicallyspaced-apart at the grating period.
 64. The diffraction grating assemblyof claim 59, wherein a ratio of the grating period to the pixel pitchalong the grating axis is substantially equal to two.
 65. Thediffraction grating assembly of claim 59, wherein the diffractiongrating is one of a plurality of diffraction gratings of the diffractiongrating assembly, the plurality of diffraction gratings being arrangedin a two-dimensional grating array disposed over the pixel array. 66.The diffraction grating assembly of claim 65, wherein the plurality ofdiffraction gratings comprises multiple sets of diffraction gratings,the grating axes of the diffraction gratings of different ones of thesets having different orientations.
 67. The diffraction grating assemblyof claim 66, wherein the multiple sets of diffraction gratings comprisea first set of diffraction gratings and a second set of diffractiongratings, the grating axes of the diffraction gratings of the first setextending substantially perpendicularly to the grating axes of thediffraction gratings of the second set.
 68. The diffraction gratingassembly of claim 59, wherein each diffraction gratings includes betweentwo and ten repetitions of the grating period.
 69. A method of capturinglight field image data about a scene, the method comprising: diffractingan optical wavefront originating from the scene with a diffractiongrating having a grating period along a grating axis to generate adiffracted wavefront; and detecting the diffracted wavefront as thelight field image data with a pixel array comprising a plurality oflight-sensitive pixels disposed under the diffraction grating, the pixelarray having a pixel pitch along the grating axis that is smaller thanthe grating period.
 70. The method of claim 69, further comprisingspatio-spectrally filtering the diffracted wavefront with a color filterarray prior to detecting the diffracted wavefront with the plurality oflight-sensitive pixels.
 71. The method of claim 69, wherein diffractingthe optical wavefront originating from the scene comprises diffractingthe optical wavefront in a waveband ranging from 400 nanometers to 1550nanometers.
 72. The method of claim 69, further comprising selecting thegrating period in a range between 1 micrometer to 20 micrometers. 73.The method of claim 69, further comprising providing the diffractiongrating as a binary phase grating comprising a series of ridgesperiodically spaced-apart at the grating period, interleaved with aseries of grooves periodically spaced-apart at the grating period. 74.The method of claim 73, further comprising providing the diffractiongrating with a duty cycle of about 50% and positioning eachlight-sensitive pixel under and in alignment with either a correspondingone of the ridges or a corresponding one of the grooves, or under and inalignment with a transition between a corresponding one of the ridgesand a corresponding adjacent one of the grooves.
 75. The method of claim73, further comprising setting a step height of the ridges relative tothe grooves to control an optical path difference between adjacent onesof the ridges and grooves.
 76. The method of claim 69, furthercomprising setting a separation distance between the refractive indexmodulation pattern of the diffraction grating and a light-receivingsurface of the pixel array to less than about ten times a centerwavelength of the optical wavefront.
 77. The method of claim 69, furthercomprising setting a ratio of the grating period to the pixel pitchalong the grating axis to be substantially equal to two.
 78. The methodof claim 69, further comprising providing the plurality oflight-sensitive pixels in a rectangular pixel grid defined by twoorthogonal pixel axes, and orienting the grating axis either parallel toone of the two orthogonal pixel axes or oblique to both the twoorthogonal pixel axes.
 79. The method of claim 69, further comprisingspectrally dispersing the optical wavefront prior to diffracting theoptical wavefront.
 80. A method of providing three-dimensional imagingcapabilities to an image sensor viewing a scene and comprising a pixelarray having a plurality of light-sensitive pixels, the methodcomprising: disposing a diffraction grating assembly in front of theimage sensor, the diffraction grating assembly comprising a diffractiongrating having a grating axis and a grating period along the gratingaxis, the grating period being larger than a pixel pitch of the pixelarray along the grating axis; receiving and diffracting an opticalwavefront originating from the scene with the diffraction grating togenerate a diffracted wavefront; and detecting the diffracted wavefrontwith the light-sensitive pixels.
 81. The method of claim 80, furthercomprising spatio-spectrally filtering the diffracted wavefront with acolor filter array prior to detecting the diffracted wavefront by theplurality of light-sensitive pixels.
 82. The method of claim 80, furthercomprising selecting the grating period in a range between 1 micrometerto 20 micrometers.
 83. The method of claim 80, further comprisingproviding the diffraction grating as a binary phase grating comprising aseries of ridges periodically spaced-apart at the grating period,interleaved with a series of grooves periodically spaced-apart at thegrating period.
 84. The method of claim 83, further comprising providingthe diffraction grating with a duty cycle of about 50% and positioningthe diffraction grating assembly over the pixel array such that eithereach ridge and each groove extends over and in alignment with acorresponding one of the light-sensitive pixels, or each transitionbetween the interleaved ridges and grooves extends over and in alignmentwith a corresponding one of the light-sensitive pixels.
 85. The methodof claim 83, further comprising setting a step height of the ridgesrelative to the grooves to control an optical path difference betweenadjacent ones of the ridges and grooves.
 86. The method of claim 80,wherein disposing the diffraction grating assembly in front of the imagesensor comprises positioning the diffraction grating assembly at aseparation distance from the pixel array selected such that an opticalpath length of the diffracted wavefront prior to being detected with thelight-sensitive pixels is less than about ten times a center wavelengthof the optical wavefront.
 87. The method of claim 80, further comprisingsetting the grating period equal to substantially twice the pixel pitchalong the grating axis.
 88. The method of claim 80, wherein disposingthe diffraction grating assembly in front of the image sensor comprisesorienting the grating axis either parallel to one of two orthogonalpixel axes of the pixel array or oblique to both the two orthogonalpixel axes.
 89. The method of claim 80, further comprising spectrallydispersing the optical wavefront prior to diffracting the opticalwavefront.